Experiment | Automatic sync light/displacement to audio

@will.day created a great animation here with Aiwa Exos 10 speaker here: Aiwa Exos 10 Speaker and it made me wonder if you could also automatically sync certain things to the audio.

I’ve done a little test and while it’s very far from perfect, the idea itself might give others some ideas on how to use this in their animations.

Preperation

I transformed a mp3 to a midi file using a free online transformer. For some reason I can’t find a way to use a mp3/wav directly which would be easier.

I first split the voice/drums/bass/others and used only the bass for this. That’s not really needed, but I wanted a bit simpler midi file for the test

In DaVinci Resolve Studio I’ve added the actual MP3 as reference and created a Fusion composition with basically a rectangle.

That rectangle has a ‘level’ and that level I connected with a right click to a midi-extractor. Based on the settings at the right the opacity of the rectangle will changed if the conditions are met. In this example a certain note range.

After export that will give you the clip as below with a flashing white rectangle which will be used as a driver for KeyShot later on.

Since I wanted to move a speaker I created a displacement map. As you will see I made a mistake and the result in the center of the subwoofer looks miserable, but hey, it’s the main concept what counts :wink:

Inside KeyShot

While I had some different ideas the options to use this clip are a bit limited by the limitations of KeyShot. For example, you can’t use it connected to a color to number node and attach it to the power input of a light.

So I kept things pretty simple using the video map with all frames of the clip in the material of the emissive LED and the displacement of the actual subwoofer.

The material of the LED:

The material of the subwoofer:

The result:

If you came till here reading, nice! Don’t ask me what the usage cases are for this, don’t have a clue. Just wanted to share a little test and maybe someone feels like doing the same but with bass//mid/treble in one video file and use it to drive a fine vintage lighted dance floor, I don’t know.

Oh, and if you experiment with it, be sure to use the same video fps in both KS as DaVinci.

3 Likes

I’m catching up from vacation… thanks for taking the time to write all this out, @oscar.rottink!

Welcome back! I’m not sure how useful it is but it was nice to try it as a little experiment. I always learn something from it.